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Essential Organizational Supports 
for Early Education 

Decades of evidence indicate that high-quality early education can positively 
affect the learning trajectories of disadvantaged young children. Accordingly, 
the early education field has made substantial investments to improve what 
occurs inside the classroom. While great strides have been made in providing 
supportive classroom environments for young children, instructional quality 
in many preschool programs remains persistently low . 1 With such an intensive 
focus on improving quality, why do we not see improvements in teaching 
and learning? 


Ample evidence displays significant achieve- 
ment gaps between more- and less-disadvantaged 
children by the time they enter kindergarten . 2 
Research also shows that high-quality early 
education programs have the potential to prevent 
or reduce those achievement gaps when children 
are three and four years old . 3 This evidence has 
garnered surprising levels of bi-partisan political 
support for significant increases in investments 
in early education programming, to develop sys- 
tems of standards and monitoring, and to expand 
professional development opportunities for early 
childhood professionals . 4 To date, efforts to pro- 
vide high-quality early education have primarily 
focused on improving what occurs within the class- 
room itself— how the classroom is set up and the 
particular interactions that take place between 
teachers and children to support social, emotional, 
and cognitive development. The paradox for the 


field is that despite these efforts to improve what 
occurs in the classroom, publicly-funded pre- 
kindergarten (pre-k) programs continue to display 
instructional quality that is too weak to prepare 
children for kindergarten . 5 

We propose broadening the focus of improve- 
ment efforts beyond the classroom and considering 
the organizational conditions that either support or 
hinder the work of teachers and the relationships 
among staff, children, and families . 6 Key here is the 
recognition that classroom processes do not occur 
in isolation from organizational processes. High- 
quality teaching and sustained child engagement 
within the classroom depends, in large measure, 
on whether leadership and staff engage in a culture 
of ongoing support and development. Attending to 
these organization-level processes— in addition to 
other specific practices within the classroom— has 
been shown to enhance the day-to-day work of 


1 Burchinal, Vandergrift, Pianta, & Mashburn (2010); 
Office of Head Start, Administration for Children & 
Families, U.S. Department of Health & Human Services 
(2013, 2014, 2015a). 

2 Denton & West (2002); Fryer & Levitt (2006); 
Magnuson, Meyers, Ruhm, & Waldfogel (2005); 

Puma, Bell, Cook, & Heid (2010). 

3 e.g., Frede, Jung, Barnett, Lamy, & Figueras (2007); 

Gormley, Gayer, Phillips, & Dawson (2005); Institute 


of Medicine & National Research Council (IOM & NRC, 
2015); Weiland & Yoshikawa (2013). 

4 Atchison & Workman (2015); IOM & NRC (2015); 
Yoshikawa et al. (2013). 

5 Burchinal et al. (2010); Office of Head Start, Admin- 
istration for Children & Families, U.S. Department of 
Health & Human Services (2013, 2014, 2015a). 

6 Regenstein, Connors, & Romero-Jurado (2016); 
Regenstein & Romero-Jurado (2014). 
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teachers, improving classroom instruction in sus- 
tained ways that lead to better student outcomes. 
Indeed, research on school improvement from 
the University of Chicago Consortium on School 
Research (UChicago Consortium) indicates that 
improving schools requires coherent, orchestrated 
action across the following five organizational 
dimensions: Effective leaders, collaborative teach- 
ers, involved families, supportive environment, 
and ambitious instruction. 7 However, early educa- 
tion has yet to incorporate these organization-level 
dimensions into definitions of quality and consider 
how the processes supporting them contribute to 
overall improvements in instructional quality. 

New Surveys Help Broaden the 
Definition of Quality 

UChicago Consortium and the Ounce of Prevention 
Fund (Ounce) are taking a first step in attempting 
to broaden the definition of quality by developing 
a set of surveys to measure key organizational 
conditions in early childhood settings. We are 
leveraging UChicago Consortium’s existing orga- 
nizational framework and set of surveys used in 
school settings— called the 5Essentials— to define 
and develop th e Five Essentials— Early Education. 
These early education surveys are being designed 
for use in publicly-funded programs— including 
federally-funded Head Start, state-, and locally- 
funded programs— that provide center-based 
services (not home-based services) to three- and 
four-year-old children. 8 Recognizing the impor- 
tance of strong organizational supports and being 
able to adequately measure them has the potential 
to shift the early education field toward more com- 
prehensively defining high-quality practices; using 
these more holistic definitions of quality may in 
turn enhance continuous improvement efforts to 


increase kindergarten readiness for all children. 

There are three phases to our survey develop- 
ment work: (1) Initial survey development and 
pilot testing, (2) a validation study, and (3) initial 
survey implementation in centers and schools. 

This brief documents the first phase of work to 
develop and pilot test the Five Essentials— Early 
Education surveys. We find it imperative to initiate 
the conversation about organizational supports 
now, as cities, states, and the nation are focusing 
intently on improving the quality of early educa- 
tion for all children, rather than waiting until we 
complete all phases of our work. Therefore, in this 
document, we share the framework underlying the 
surveys, describe our survey development process, 
define the specific constructs measured by the 
surveys, and present initial survey reliability 
results from a pilot study conducted in spring 
2015. Over the next year, we will complete our 
validation study and then transition into shap- 
ing implementation and supporting the use of the 
surveys within pre-k settings. 

We present this brief to practitioners and 
researchers who are interested in measuring the 
quality of organization-level supports. For all 
audiences, this brief aims to explain why the early 
education field would benefit from a measurement 
system that captures the strength of organizational 
processes. By providing concrete definitions of the 
organizational conditions being measured by our 
new surveys, we encourage practitioners to begin 
conceptualizing what this may mean for their own 
work. F or other researchers seeking to develop 
new surveys, this brief provides a roadmap of our 
rigorous survey development process, whereby we 
describe our methods for achieving reliable and 
valid measurement of our intended constructs. 

Ultimately, the Five Essentials— Early Education 


7 Bryk (2010). 

8 Programs can be located within schools (which we 
refer to as ‘'school-based”) or within a community- 
based setting (which we refer to as “center-based”). 
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surveys are designed to capture indicators of pro- 
cesses occurring at the school or center level that, 
in prior research, have been linked to improve- 
ments in student learning. 9 As such, they aim to 
take the temperature of a program so that leaders 
and staff can collaborate on identifying the pro- 
cesses behind what is working well and what may 
need improvement. We intend for these surveys 
to highlight what strong organizational supports 
are, encouraging the early education held toward 
a more comprehensive definition of high-quality 
pre-k. On the ground, we aspire to provide school 
and center leaders with reliable and valid data 
on key organizational conditions that reinforce 
positive instructional practices among teachers 
and staff as they work with preschool children 
and families. 

Early Education Programs 
Can Contribute to Closing the 
Achievement Gap, But Many Do 
Not Meet Expectations 

The great emphasis on early education in the U.S. 
is driven by evidence that low-income, high-needs 
children enter kindergarten significantly behind 
their better-resourced peers, 10 and that gaps in 
early academic skills continue to persist or even 
widen in the elementary years. 11 For example, na- 
tional data from the Early Childhood Longitudinal 
Study-Kindergarten Cohort (ECLS-K) found a 
difference of one full standard deviation (or 15 
standard score points) in literacy and mathematics 


between children from low- and high-income fami- 
lies at the beginning of kindergarten. 12 In addition, 
children from lower-resourced families commonly 
have not developed the age-expected self-regu- 
lation and social-emotional skills necessary for 
navigating K-3 classrooms, which may limit their 
capacity for learning in these environments. 13 

A substantial body of research suggests that 
high-quality preschool can help to narrow these 
gaps. Historically, intensive programs including 
Perry Preschool, Abecedarian, and Child-Parent 
Centers have shown long-term benefits for par- 
ticipating children. 14 And collective understand- 
ing of how to support the early learning of young 
children and their families has led to substantial 
positive changes to the ever-growing field over the 
last several decades. There have been considerable 
investments in establishing quality standards and 
providing professional development to improve 
practices toward those standards. Recently, fed- 
eral Head Start accountability policies and some 
state accountability policies have shifted from 
focusing solely on structural elements of quality 
(for example, how the room is arranged for play, 
as measured by the Early Childhood Environment 
Rating Scale [ECERS]) 15 to ones that incorpo- 
rate how teachers and children interact with each 
other (as measured by the Classroom Assessment 
Scoring System Pre-K [CLASS Pre-K], 16 described 
more below). 17 Although development ally- 
appropriate structural components are necessary 
for high-quality early education, the inclusion of 


9 Bryk, Sebring, Allensworth, Luppescu, & Easton (2010). 

10 Denton & West (2002); Magnuson et al. (2005); Puma 
etal. (2010). 

11 Fryer & Levitt (2006); Loeb & Bassok (2007); Princiotta, 
Flanagan, & Gernimo-Hausken (2006). 

12 Zill& West (2001). 

13 Blair & Razza (2007); Fantuzzo, Bulotsky-Shearer, 
McDermott, & McWayne, (2007); Reardon (2011). 

There is also emerging evidence that some sub- 
populations of children may be faring better than 
others. Research points to what some have called 
the "immigrant paradox;” this phenomenon refers to 
young children of recent immigrants who show similar 


or stronger self-regulation and social-emotional skills 
than native English speaking children, although these 
differences often disappear when accounting for other 
background characteristics (Halle et al., 2014). 

14 e.g., Campbell, Ramey, Pungello, Sparling, & Miller- 
Johnson (2002); Heckman, Moon, Pinto, Save-lyev, 

& Yavitz (2010); Reynolds, Temple, White, Ou, & 
Robertson (2011); Schweinhart, Montie, Xiang, Barnett, 
Belfield, & Nores (2005). 

15 Harms, Clifford, & Cryer (2005). 

16 Pianta, La Paro, & Hamre (2007). 

17 The Build Initiative & Child Trends (2015); Connors & 
Morris (2015). 
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interactions and relationships among teachers and 
children in standards for quality— and support for 
improving those interactions through training and 
technical assistance— significantly advances the 
early education field. In fact, recent evidence shows 
that locations such as Boston, Oklahoma, New 
Jersey, and Tennessee are implementing quality 
state-funded pre-k programs at scale and improv- 
ing cognitive outcomes for low-income, high-needs 
children by as much as one-third to three-quarters 
of a standard deviation, compared to similar chil- 
dren in control groups. 18 Often, these programs 
use research-based curricula that include a focus 
on cultural and linguistic responsiveness, and 
provide teachers with coaching supports. 19 

While individual programs have shown prom- 
ise, high-quality instruction does not currently 
exist across programs at scale. 20 In recent years, 
the most common tool used to measure classroom 
interactions among teachers and children is the 
CLASS Pre-K. 21 Classrooms are observed and 
scored on three “domains”— Emotional Support, 
Classroom Organization, and Instructional 
Support. 22 Each domain is scored on a 7-point 
scale; classrooms scoring between 1 and 2 are 
considered to be low-quality, those scoring a 3, 4, 
or 5 are considered mid-level quality, and those 
scoring a 6 or 7 are considered high-quality. 
Observations from publicly-funded programs 


across the country indicate that although many 
teachers provide high-quality emotional and 
classroom organizational supports to children, 
the quality of instructional supports in many 
programs is often too low to influence children’s 
learning. 23 For example, in 2015, a large, national 
sample of Head Start classrooms received an aver- 
age score of 6.03 on Emotional Support and 5.80 
on Classroom Organization. 24 In contrast, those 
same classrooms received an average score of 2.88 
out of 7 points on Instructional Support— in the 
low-mid quality range. And the quality of instruc- 
tional supports remained stagnant year after year, 
with averages of 2.90 in 2014 and 2.72 in 2013. 25 
Classrooms at these levels display teacher-child 
interactions that contain too little concept de- 
velopment support, instructional feedback, and 
advanced language to impact children’s readiness 
for kindergarten. 26 Indeed, the ongoing weakness 
in instructional support is evident in the continu- 
ing achievement gap that exists when children 
enter kindergarten. 27 

Explanations for this stubborn underperfor- 
mance on instruction across many pre-k programs 
include a wide range of variables, including: 

Family, community, and child risk factors; 28 low 
classroom structural quality; 29 incoherent un- 
derstanding and implementation of instructional 
strategies; 30 and teacher compensation, work 


18 Frede et al. (2007); Gormley et al. (2005); Weiland & 
Yoshikawa (2013). 

19 Garcia & Frede (2010); Weiland & Yoshikawa (2013); 
Yoshikawa et al. (2013). 

20 Yoshikawa et al. (2013). 

21 Pianta et al. (2007). 

22 Each of these three ''domains” are comprised of 
several "dimensions.” Emotional Support is comprised 
of Positive Climate, Negative Climate, Teacher Sensi- 
tivity, and Regard for Student Perspective. Classroom 
Organization is comprised of Behavior Management, 
Productivity, and Instructional Learning Formats. 
Instructional Supports is comprised of Concept 
Development, Quality of Feedback, and Language 
Modeling. See Pianta et al. (2007) for full descriptions. 

23 Aikens, Klein, Tarullo, & West (2013); Barnett, Carolan, 
Squires, Clarke Brown, & Horowitz (2015); Burchinal et 
al. (2010); Early et al. (2007). 


24 Office of Head Start, Administration for Children & 
Families, U.S. Department of Health & Human Services 
(2015a). We focus here on Head Start programs simply 
because the data are available for them. Similar data 
do not exist for other publicly-funded pre-k programs 
across the country, although individual studies have 
indicated similar trends (see FNs 17 and 18). 

25 Office of Head Start, Administration for Children & 
Families, U.S. Department of Health & Human Services 
(2013, 2014). 

26 Burchinal et al. (2010). 

27 Mulligan, Hastedt, & McCarroll (2012). 

28 e.g., Brooks-Gunn, Johnson, & Leventhal (2010); 
Garrett-Peters, Mokrova, Willoughby, & Pan (2016). 

29 e.g.. La Paro, Thomason, Lower, Kintner-Duffy, & 
Cassidy (2012). 

30 e.g., Barnett (2003); Howes (2010); IOM & NRC (2015). 
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conditions, and job satisfaction . 31 However, the 
held has yet to seriously consider the relationship 
between classroom processes and the higher- 
level organizational processes that can influence 
whether practices inside the classroom improve 
or stagnate. While each of the aforementioned 
factors is likely part of the explanation for low 
levels of instructional quality, they ignore the 
complex, integrated organizational structures 
that actively support the continuous work of 
teachers and the conditions under which those 
teachers engage with each other, children, curricu- 
lum, and families . 32 In other words, we maybe 
losing sight of the forest for the trees. 

The opportunity to create fundamental, 
systemic improvements in pre-k instruction 
may lie in broadening our understanding of 
high-quality early education. This suggests 
moving the focus beyond individual pieces of 
structural and classroom quality to the organi- 
zational conditions that underlie high-quality 
practice and improvement more broadly . 33 This 
project aims to take the first step in helping to 
broaden the focus by defining these organizational 
conditions, based on theory and prior empiri- 
cal work, and developing a valid and reliable way 
to measure them. Toward this end, UChicago 
Consortium and the Ounce are developing the 
Five Essentials— Early Education surveys for use 
in early education settings. Ultimately, the pur- 
pose of the Five Essentials— Early Education proj- 
ect is to provide actionable survey data on broader 
organizational constructs that may be a crucial 
piece of the early education quality puzzle. 


The Five Essentials— Early 
Education Provides a Framework 
and Tools to Measure Key 
Organizational Conditions 

The field of early education has a history of 
measuring classroom-specific structural compo- 
nents and evidence-based classroom practices. 
There are, for example, reliable and valid tools to 
measure classroom structural quality, classroom 
interactions and instruction, interactions with 
families, administrative practices, and program 
climate . 34 These measures are helpful in identify- 
ing some areas for improvement within programs, 
but none measure the organizational conditions 
that support implementation of those structural 
components and practices . 35 The Five Essentials— 
Early Education fills this critical measurement 
gap. Being able to identify and measure programs’ 
organizational conditions can allow leaders to 
focus on strengthening the supports staff need to 
successfully improve their work. 

Research In Elementary Schools Provides 
a Way to Think About Key Organizational 
Conditions In Early Education Settings 

Prior research on elementary schools conducted 
by UChicago Consortium researchers led to the 
identification of five school-level organizational 
supports that work together to support school 
improvement, articulated in the five essential 
supports framework. 36 These five constructs, 
defined in more detail below, are: effective leaders, 
collaborative teachers, involved families, sup- 
portive environment, and ambitious instruction . 37 


31 e.g., Whitebook & Ryan (2011); Whitebook, Phillips, & 
Howes (2014). 

32 Bryk et al. (2010); Zaslow, Tout, & Martinez-Beck (2010). 

33 Bouffard& Jones (2011). 

34 e.g., Bryant (2010). Examples of tools measuring these 
discrete constructs are: the Preschool Program Quality 
Assessment (PQA; High/Scope Educational Research 
Foundation, 2003), the Program Administration Scale 
(PAS; Talan & Bloom, 2004); CLASS PreK (Pianta et 
at, 2007); ECERS (Harms et al., 2005); the Child Care 


Worker Job Stress Inventory (CCWJSI; Curbow, Spratt, 
Ungaretti, McDonnell, & Breckler, 2000), the Early 
Childhood Job Satisfaction Survey (ECJSS; Bloom, 
2010); and the Early Childhood Work Environment 
Survey (ECWES; Bloom, 2010). 

35 Zaslow et al. (2010). 

36 Bryk et al. (2010). 

37 The original study (Bryk et al., 2010) used different 
language than what is presented here, but the terms 
have since been adapted for easier interpretation. 
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The conceptual framework (see Figure 1) specifies 
these five essential supports as a set of subsys- 
tems that operate in strong cooperation with one 
another to either enable or inhibit improvement in 
classroom practices. 

Each essential has unique salience within 
the organization, but they are mutually reinforc- 
ing. In fact, when one of the essential supports is 
strengthened, it buttresses the development of the 
other supports. For example, improving school 
culture in ways that increase collaboration among 
teachers is likely to also lead to higher- quality in- 
struction and a safer learning space for students— 
that is, strengthening ambitious instruction and 
supportive environments. In turn, teachers are 
more likely to rely on one another for ideas on how 
to provide better instruction. They may also gen- 
erally feel more supported, freeing themselves up 
to better engage with their students in ways that 
speak to students’ individual needs— academic or 
otherwise. In other words, these five essential sup- 
ports operate in tandem, influencing not only each 
other, but ultimately the professional relation- 
ships, capacity-building, classroom interactions, 
teaching, and learning that occur within a school. 
School leadership is theorized to be the driving 
force, acting as the catalyst for improvement by 
facilitating coherent change and strengthening 
the other four essentials. 

Bryket al. (2010) originally developed the 
5Essentials surveys for teachers in all elementary 
grades (K-8) and students in grades 6 and above, 
with expansion to high schools in later years to 
measure these five organizational conditions. They 
found that responses to the surveys strongly pre- 
dicted which schools were most and least likely 
to improve over time: elementary schools strong in 
three or more of these supports were 10 times more 


FIGURE 1 

The Five Essential Supports Influence 
Classroom Practices 



likely than schools weak in most supports to sub- 
stantially improve student math and reading test 
scores over the next few years. Furthermore, it was 
unlikely that a school had a sustained weakness in 
just one of these supports over time; however, in 
the rare cases when that occurred, the weakness in 
one domain ultimately undermined other change 
efforts. Fewer than 10 percent of schools with a 
sustained weakness in one essential support 
showed improvements in student outcomes. 38 

We Are Developing New Surveys to Identify 
the Strength of these Five Organizational 
Conditions in Early Education Settings 

While the original five essential supports frame- 
work was developed with elementary schools in 
mind, research in early education likewise indicates 
that programs are more successful in promoting 
children’s learning when they have strong facilita- 
tive leadership and organizational structures in 


38 Bryk et al. (2010). 
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place to support teachers’ practices and ongoing 
learning. 39 Therefore, we posit that pre-k schools 
or centers that are strong across these five es- 
sential supports— indicating a cohesive environ- 
ment where teachers feel supported, leaders are 
focused on supporting improvements in instruc- 
tion, and children are nurtured both cognitively 
and emotionally— will also display higher- quality 
classroom interactions and better child outcomes. 
In order to test whether these relationships exist, 
we must first measure these constructs in early 
education settings. This starts with an articula- 
tion of what the five essential supports would look 
like in early education. Here, we provide those 
definitions. They are heavily based on Bryk et al.’s 
original five essential supports definitions, with 
adaptations to improve their fit with the goals and 
language of the early education field. 40 

Effective Instructional Leaders: The school or 
program leadership is strategically focused on 
children’s development and early achievement. 
They nurture trust, collective understanding and 
responsibility for excellence, and improvement 
among staff and families. 

Collaborative Teachers: Teachers are committed 
to the school or program, build strong relation- 
ships with their colleagues, and work together 
continuously to improve teaching and learning by 
engaging in research-based, data-informed cycles 
of professional development. 


Involved Families: Staff develop strong relation- 
ships with families and support active family 
engagement in children’s learning. 

Supportive Environments: Schools or programs 
are physically and emotionally safe and engaging 
environments, wherein staff hold high expecta- 
tions for children’s social-emotional and academic 
learning, coupled with nurturing, individualized 
support for children and families. 

Ambitious Instruction: Teachers and staff provide 
consistently engaging, effective, rigorous, and 
developmentally-appropriate curriculum and 
instruction. 

The Development of the Five 
Essentials— Early Education 
Surveys 

The development and pilot testing of the Five 
Essentials— Early Education surveys has taken 
two years to complete. We designed the Five 
Essentials— Early Education as a single tool that 
comprehensively measures the five organizational 
constructs defined above. While the existing 
K-12 5Essentials surveys tap into these ideas 
through surveys available to teachers and stu- 
dents, the early education version measures these 
constructs through the perceptions and experi- 
ences of teachers and parents. 41 We drew heavily 
from UChicago Consortium’s existing 5Essentials 
teacher survey, which has been in use for over 


39 Lower & Cassidy (2007); Muijs, Harris, Chapman, Stoll, 
& Russ (2004); West-Olatunji, Behar-Horenstein, & 
Rant (2008). 

40 For instance, the K-12 definition of Supportive 
Environment is: “The school is safe and orderly. 
Teachers have high expectations for students. 
Students are supported by their teachers and peers.” 
The early education version expands on those ideas 
to emphasize that high expectations are for both 
social-emotional and academic development. It also 
includes a focus on providing supports to families in 
addition to children. Another language adaptation 
was changing “Effective Leaders” to "Effective 
Instructional Leaders” to highlight the importance of 
instructional leadership as distinct from management. 


41 See http://uchicagoimpact.org/5essentials for more 
information on the use of the K-12 5Essentials in sur- 
veys across the country. See also http://consortium. 
uchicago.edu/surveys for more information on the 
surveys themselves. Also note that while we use the 
term “parent survey,” we intend for them to be filled 
out by any adult caregiver engaged in their child’s 
pre-k experiences. In addition, we are testing the 
teacher survey with other pre-k staff, not solely 
teachers. However, most questions ask about what 
teachers in the school or program do. 
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20 years with ongoing refinement, to develop the 
Five Essentials— Early Education teacher survey. 
Because there was no parent survey to draw from, 
the Five Essentials— Early Education parent survey 
was created as a new survey through the survey 
development process, described below. Parents 
are a key stakeholder in the early learning space; 
they are able to provide a unique perspective on 
the ways in which early education programs are, 
or are not, structured to support their child and 
their family. Together, the teacher and parent 
surveys are designed to provide a well-rounded 
picture of how early education program leaders 
support teachers and other staff and, in turn, how 
staff support children and families toward early 
learning goals. 

To develop the new Five Essentials— Early 
Education surveys, we engaged in an iterative 
development process with several rounds of cre- 
ation, testing, analysis, and revision. We focused 
our development and testing on publicly-funded 
early education programs serving children ages 


3 to 5 in either school- or center-based settings. 42 
Qualitative and quantitative feedback from one 
cycle was used to inform the next phase of develop- 
ment and testing. Our method of survey develop- 
ment includes the creation of “measures,” each 
of which each represents a particular construct 
(such as trust among teachers) made from corre- 
sponding questions (or survey items). Responses 
to questions within each measure are analyzed 
using the Rasch IRT model. 43 Rasch theory 
posits that questions of varying degrees of diffi- 
culty (in this case, ease or difficulty to endorse 
or agree with ) differentiate people’s placement 
along a developmental scale: Endorsing more 
difficult questions means the respondent has 
higher levels (or more positive beliefs) on the 
underlying construct. Our use of questions and 
measures is modeled after the K-12 5Essentials 
surveys. Figure 2 shows howthe Effective Leaders 
essential is constructed by four measures. In turn, 
it shows how one of those measures, Program 
Coherence, is comprised of survey questions. 


FIGURE 2 

Similar to the K-12 5Essentials Surveys, the Five Essentials— Early Education Surveys include 
Questions that Are Combined into Measures Capturing Beliefs and Experiences of Respondents 


5Essentials 



Measures 


Instructional 

Leadership 

Teacher 

Influence 


Program 

Coherence 


< 


Teacher- 

Principal 

Trust 


Questions (Survey Items) 

To what extent do you disagree or agree 

with the following? (strongly disagree, disagree, 

agree, strongly agree) 

• Once we start a new program, we follow 
up to make sure it’s working. 

• We have so many different programs in this 
school that I can’t keep track of them all. 

• Many special programs come and go 
at this school. 

• Curriculum, instruction, and learning materials 
are well coordinated across different grade 
levels at this school. 

• There is consistency in curriculum, instruction, 
and learning materials among teachers in the 
same grade level at this school. 


42 This is our key audience because these programs 
serve children who are most at risk for entering 
kindergarten without the necessary academic and 
social-emotional skills; therefore, these programs 


have the greatest potential for reducing or closing the 
achievement gap. 

43 Wright & Masters (1982). 
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The development and testing of both the 
teacher and parent surveys included steps to 
determine what the surveys should include, 
whether our items were appropriately measur- 
ing those constructs, and ultimately whether the 
surveys themselves were reliable. Here, we take 
you through our development process. Because 
the parent survey was created anew and we could 
not draw on an existing K-12 version, it required a 
more nuanced and iterative development process. 
Therefore, steps were repeated along the way as we 
continuously refined the questions and measures 
for the parent survey. Below, and in Figure 3, we 
take you more specifically through these three 
steps of our development process. 

Step 1: Determine Content 

The first step in our development process was to 
determine which topics should be included on 
each survey. Theoretically we were interested in 
capturing aspects of early education programs 
that related to the five essential supports defined 
on page 7. To develop the teacher survey, we relied 
heavily on the existing K-12 5EssentiaIs teacher 
survey developed by UChicago Consortium. 44 The 
K-12 5Essentials surveys have been administered 
since 1994, resulting in a large bank of teacher sur- 
vey questions and measures from which to choose. 
Ultimately, our selection of survey questions to 
pilot for early education settings was informed 
by our collective expertise in early education 
programming, practice, and research. We deter- 
mined whether existing teacher questions fit into 
one of three categories: (1) Questions that were 
appropriate as-is for the early education context, 
(2) questions that needed terminology revisions 
for early education, and (3) questions that were 
irrelevant to early education and, therefore, could 


be discarded. Then, we determined whether there 
were concepts that are critical in early education 
but not addressed in the existing K-12 5Essentials 
surveys, and generated new questions to capture 
these concepts. For example, the K-12 5EssentiaIs 
survey does not ask teachers about parent influ- 
ence in decision-making at the school; however, in 
early childhood it is often an explicit goal to include 
parent voice in program and policy decisions. 45 
Therefore, we created a new measure attempting 
to capture this aspect of a pre-k program. 

As we developed the parent survey, we took 
additional steps to ensure it included topics that 
not only aligned with the five essential supports 
framework, but also represented what parents 
had described as most important to them and 
for which they felt they could provide accurate 
perspectives. Therefore, to determine appropri- 
ate content, we conducted focus groups with 
various stakeholders, including early education 
family engagement staff as well as parents who 
were involved in their program’s Head Start Policy 
Council, Parent Committee, Local School Council, 
or local community organizations. Conducting 
focus groups helped us identify key concepts to 
include on the survey and provided examples 
of terminology that would evoke the intended 
meaning of our questions. For example, on the 
K-12 5EssentiaIs surveys, ambitious instruc- 
tion is measured by asking students about their 
experiences in the classroom (e.g., "In your math 
class , how often do you apply math to situations in 
life outside of school?’"). Initially, we planned to 
capture ambitious instruction by asking parents 
to report on the types of instructional activities 
occurring in their child’s classroom. However, our 
focus groups revealed that parents are not com- 
fortable reporting on the specific instructional 


44 University of Chicago Consortium on School Research 45 Head Start Resource Center (2011); Office of Head 
(2014). See http://consortium.uchicago.edu/surveys Start, Administration for Children & Families, U.S. 

for more information. Department of Health & Human Services (2015b). 


Survey Development Brief 


9 


FIGURE 3 

Timeline of Five Essentials— Early Education Survey Development and Testing 


Review K-12 5Essentials Teacher Survey 

We reviewed existing survey questions and asked: 

• What questions are appropriate for use as-is in 
early education? 

• What terminology needs to be revised for use in 
early education? 

• Which questions are irrelevant or inappropriate 

for use in early education (and are not to be used)? 

• What other concepts that are critical in early 
education are missing and need to be developed? 


Teacher Survey 



2014-15 





Step 1: 

Step 2: 


Determine 

Ensure 


Content 

Content Validity 


Parent Survey 




New Survey 
Question 
Development 




Focus Groups with Parents and 
Family Engagement Staff 

Conducting focus groups helped us identify key 
concepts to include on the parent survey and pro- 
vide examples of terminology that would evoke the 
intended meaning of our questions. During our focus 
groups we asked the following types of questions: 

• What do parents care more about in their child’s 
early education program? 

• How do parents determine whether a program is 
high quality? 

• What makes parents feel connected to their 
child’s early education program? 

• What do parents want to be asked about on a 
survey of their child’s program? 


Cognitive Interviews with Parents 

Cognitive interviewing is a method of pre-testing, 
involving one-on-one interviews with individuals in 
the target population. Respondents are asked survey 
questions in a semi-structured format to explore 
their thought processes and challenges answering 
each question. The cognitive interviews helped us 
answer the following questions: 

• Are the questions and response categories easy 
to understand? 

• Do parents interpret our survey questions the 
way we intended? 

• Do the response categories appropriately capture 
parents’ experiences? 
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Pilot Testing and Analysis 

Small field pilots allowed us to test how teachers 
and parents would respond to our survey questions. 
We then used Rasch analysis to understand: 

• Are the survey measures reliable? (Reliability) 


Do they differentiate among teachers and 
parents who have different levels of belief or 
experiences, particularly at the top and bottom 
of the distribution? (Precision) 


Do responses to questions within a measure 
match the conceptual difficulty of those 
questions? (Construct Validity) 

Do questions within a measure tap into the same 
underlying construct? (Internal Validity) 



r 



r 

lot Testin 

and 

19 1 


Ana lysis 




Iterative Development Process 
for Parent Surveys 

The development of new surveys is an iterative process, 
including significant input from stakeholders. To design 
the new parent survey, we first completed a full cycle of 
steps 1-3 described in the arrow above, including a small 
pilot in schools and centers. Analyses from the small pilot 
led to a second cycle through steps 1-3, including revi- 
sions to the survey questions and measures, additional 
cognitive interviews, and a final round of pilot testing. 
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interactions taking place in the classroom. While 
they were confident their child was engaged in 
their pre-k program and learning (i.e., their child 
may come home singing a new song or recogniz- 
ing a letter the child did not know before), parents 
were not able to report on whether high-quality 
instructional interactions took place within the 
classroom in support of this learning. Because of 
what we heard from parents in our focus groups, 
we decided to develop questions that captured am- 
bitious instruction for the teacher survey instead. 
In another example, we created a measure called 
Teacher Care and Responsiveness towards Parents 
based on our focus groups. It was clear from our 
discussions that parents feel most comfortable 
with a program when teachers and staff not only 
attend to and care about their child, but also at- 
tend to the needs, struggles, and accomplishments 
of the parents themselves. 

Step 2: Ensure Content Validity 

Most of the teacher survey was selected, or adapted, 
from the K-12 5Essentials teacher survey. Even in 
the case of the newly-developed ambitious instruc- 
tion questions, the survey items were modeled after 
those that had been used with elementary teach- 
ers in prior iterations of the 5Essentials survey. 
Therefore, we had confidence that teachers would 
understand the questions and that they would elicit 
the concepts intended and, thus, decided not to 
conduct additional testing with those questions. 

However, the parent survey required an extra 
step to ensure we had content validity— that our 
questions were worded appropriately to portray 
what we wanted to ask about. In other words, we 
wanted to know: Do the questions on our survey 
capture the concepts— based on parents’ inter- 
pretations of the survey questions— they were 


designed to measure? We also sought to under- 
stand how accessible particular words were to 
parents, in an effort to make the survey as easily 
understandable and reliable as possible. To answer 
these questions, we conducted cognitive inter- 
views with parents. 46 Cognitive interviewing 
is a method of pre-testing involving one-on-one 
interviews with individuals in the target popula- 
tion. 47 Respondents are asked survey questions in 
a semi-structured format to explore their thought 
processes and challenges answering each question. 
This method can be an inexpensive and effective 
way of identifying problems in surveys. Cognitive 
interviews ensure that the questions are easy 
to understand, the question stems and answer 
choices are unambiguous, and the questions asked 
are being interpreted as intended. In our develop- 
ment process, the cognitive interviews provided 
an additional opportunity to ensure content 
validity on the parent survey and led to improved 
clarity of the questions and measures. Findings 
from cognitive interviews resulted in revisions of 
questions and response categories before formally 
testing the reliability of the parent survey. One 
final assessment of the parent survey included a 
Lexile analysis to ensure that each measure was 
written at a high-school reading level or below. 

Step 3: Test for Reliability and Internal Validity 

The last step in our survey development process 
focused on determining whether the newly-devel- 
oped surveys reliably captured a range of percep- 
tions from teachers and parents. We conducted a 
small pilot with the parent survey prior to a larger, 
final pilot that also included the teacher survey. 
Each pilot was designed to answer questions 
about the psychometric functioning of the survey 
questions within measures; this process was 


46 In conducting our cognitive interviews, we followed 
methods as described in Willis (2005). 

47 Conrad & Blair (2004); Willis (2005). 


UChicago Consortium and the Ounce | Essential Organizational Supports for Early Education 


about assessing reliability, gauging various forms 
of internal validity, and ensuring that questions 
captured a range of beliefs and experiences people 
had (i.e., that our questions were not too “easy” or 
too “hard” to endorse. The final pilot of both the 
teacher and parent surveys took place in spring 
2015 and sought to answer the following questions: 

• Are the survey measures reliable? (Reliability) 

• Do they differentiate among teachers and 
parents who have different levels of beliefs or 
experiences, particularly at the top and bottom 
of the distribution? (Precision) 

• Do responses to questions within a measure 
match the conceptual difficulty of those 
questions? (Construct Validity) 

• Do questions within a measure tap into the 
same underlying construct? (Internal Validity) 

We created the surveys with the Rasch model in 
mind, paying particular attention to the whether 
or not questions within a measure were asking 
about a single construct and purposely developing 
questions that represented a range of difficulty 
for that construct. Rasch analyses then provided 
evidence on whether the measures were indeed 
operating as intended. These analyses were used 
to discern whether questions fit together properly 
within a measure and whether the measures were 
reliable. Our goal was to develop measures that 
maximize reliability while keeping the number of 
questions as low as possible, so as to not unneces- 
sarily burden survey takers. See the box titled 


Piloting the Surveys: Data Collection and Analyses 

on p.14 for more information on the analyses and 
how the results were used to ensure the surveys 
were functioning well in the field. 

Pilot Results: How did the Five 
Essentials— Early Education Surveys 
Perform in the Pilot Study? 

The results of our pilot study analyses address the 
questions in Step 3 of our development process. We 
present here the reliability coefficients, which tell 
us whether each measure had low levels of random 
error. Consistent with others who use Rasch, we 
aim for measures to have reliabilities above 0.80, 48 
but find measures above 0.70 to be acceptable at 
the individual level. 49 

Table 1 presents results for the Five Essentials— 
Early Education teacher survey. It lists each 
measure and its corresponding Rasch reliabil- 
ity after adjusting the measure based on Rasch 
output (as described in the box titled Piloting 
the Surveys: Data Collection and Analyses on 
p.14). 50 Most measures on the teacher survey (19 
of 27) had reliabilities at or above our preferred 
threshold of 0.80, and all but one measure met 
our 0.70 minimum threshold. The one measure 
with reliability below 0.70— Early Childhood 
Discipline— has been removed from the survey for 
future iterations. Measures that were retained for 
future administrations but had reliabilities below 
0.80 were revised for a new round of testing in 


48 e.g., Amin et al. (2012); Chen, Pan, Chung, & Chen (2013); 
Linacre (2015). 

49 The intended use of these measures is to explore these 
constructs at the school- or center-level, so school-level 
or center-level reliabilities are more important and are 
expected to be higher than individual reliabilities. 

50 Using output provided by Winsteps, the authors paid 
particular attention to several statistics for measure 
and item evaluation. These were our general guide- 
lines, but many of them are not absolutes and can 
be over-ridden by theoretical and practical consider- 
ations: (1) Item separation and reliability (model ver- 
sion, of non-extreme people) to ensure that reliability 
was above 0.70, preferably above 0.80 (which corre- 
sponds to a separation > 2); (2) category fit statistics 


and Andrich thresholds, to ensure that the observed 
average for each category increased monotonically; 

(3) infit mean square, to identify questions that were 
above 1.3 (considered by some to be a conservative 
cutoff, e.g., Prieto, Contador, Tapias-Merino, Mitchell, 
& Bermejo-Pareja, 2012); an indication that they are 
measuring something other than the construct being 
tapped by all other questions in the measure this 
identified questions as candidates for removal; and 

(4) item difficulty, to ensure that questions were 
ordered in a way we would theoretically expect. 

We also used item difficulty to identify redundant 
questions in terms of difficulty (i.e., not adding much 
strength to the overall measure, particularly when 
there were too many questions). 
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Piloting the Surveys: Data Collection and Analyses 


Data Collection Efforts 

Teacher Survey 

The Five Essentials— Early Education teacher 
survey was largely based on existing banks of 
questions; therefore, there was only one round 
of development and testing— addressing ques- 
tions in Steps 1 and 3 of our development process. 
The spring 2015 pilot testing occurred in both 
school- and center-based settings. To pilot in 
school-based programs, the survey was offered 
to all preschool teachers in the Chicago Public 
Schools (CPS) through an online survey adminis- 
tration tool; a total of 1,153 teachers responded. A 
Because the school-based survey was part of a 
larger survey given to teachers, the full comple- 
tion time (for all survey questions, not just ours) 
was roughly 30-40 minutes. Teacher pilot data 
were also collected from a national sample of 
center-based Head Start teachers participating in 
the Family and Child Experiences Survey (FACES, 
conducted by the Office of Planning, Research, 

& Evaluation at the Administration for Children & 
Families). 6 Eighty-eight percent of those teach- 
ers sampled responded to the survey (n=363). 
Because our items were added onto existing 
surveys for the FACES study, there were two ver- 
sions— Version A and Version B— each of which 
was given to roughly half of the participating 
teachers (Version A was taken by 177 teachers; 
Version B by 186 teachers). This kept the aver- 
age time it took to complete the survey to under 
20 minutes. Teachers in the FACES sample were 
provided the option to complete the survey online 
(68.3 percent) or using a paper-pencil version 
(31.7 percent). All teacher surveys were provided 
in English only. 

Parent Survey 

The development of the Five Essentials— Early 
Education parent survey involved two rounds 
of interviews with parents and staff and pilot data 
collection. Our first round included the following 
processes in fall 2014: Three focus groups with 
parents and one focus group with early educa- 
tion family support specialists (Step 1 of our 
development process); cognitive interviews with 
20 parents (Step 2 of our development process); 


and preliminary field testing in four center-based 
programs, resulting in over 200 surveys from 
English- and Spanish-speaking parents (Step 3 of 
our development process). Following analyses on 
the small pilot, a second round of development 
included revisions to questions and measures 
based on our Rasch analyses. We conducted ad- 
ditional cognitive interviews on newly developed 
questions with nine parents in preparation for the 
final pilot. 

The final pilot occurred in spring 2015, when 
the parent survey was administered on-site 
in seven school-based and nine center-based 
preschool sites. Surveys were collected by the 
authors and research team members. Teams of 
data collectors visited sites and recruited parents 
to take the pilot survey between May-June 2015. 
The online survey was developed using Qualtrics 
and tablets were made available to parents for 
use; hard copy versions were available upon 
request or used when difficulties arose connecting 
to the internet. Surveys were available in both 
English and Spanish. There were 253 parents who 
responded to the survey (139 in English and 114 
in Spanish, evenly distributed over school- and 
center-based sites); 81 percent were completed 
online using tablets with the remaining filled 
out using paper-pencil versions (mostly due to 
internet connection issues). On average, surveys 
took parents between 10-20 minutes to complete. 

Analyses 

Final analyses on the Five Essentials— Early 
Education surveys were conducted on the 
combined sample from CPS and FACES for the 
teacher survey and on the combined sample from 
school- and center-based programs in Chicago 
for the parent survey. All analyses were conduct- 
ed using the Rasch model, with Winsteps Rasch 
Measurement Program, Version 3.90.2. C,D 

Using Rasch Helps Ensure Reliability and 
Construct and Internal Survey Validity 
Rasch output provides abundant information 
about the functioning of our measures. Our 
analyses helped us address the questions laid out 
in Step 3 of our development process. Of primary 
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importance is that our measures were reliable: 
That what we are measuring captures the true 
response (or experience) of the respondent. In 
other words, responses to survey questions do 
not have large amounts of random error. Rasch 
reliability coefficients also helped us determine 
whether the questions in the measure have 
enough precision to differentiate across people 
who hold different opinions about a construct. 

We used these information points to decide 
whether new questions should be written to 
be more sensitive to particular portions of the 
distribution (e.g., writing questions that are 
easier or harder to endorse). 

Using information garnered from our Rasch 
analyses, we could also ascertain whether we had 
construct validity and internal validity in our survey 
items (or questions) and measures. Rasch output 
includes “item difficulty,” which indicates which 
items within a measure are easier vs. harder for 
respondents to endorse. For example, consider 
a measure about director/principal relations with 
parents. Two questions in the measure may be: 

(1) How often is the director/principal visible when 
you are in the center? and (2) How frequently has 
the director/principal reached out to get feed- 
back about your child’s experience? Because it is 
pretty common for a center director to be visible 
to parents, the first question would be considered 
“easier to endorse,” with many parents responding 
in the top response category ("daily”). In contrast, 
questions that are considered more “difficult to 
endorse” are ones where fewer people choose 
the highest response category. The second ques- 
tion here asks something that is less common 
in programs; it is less likely that parents would 
respond with the highest response category (“all 
the time"). The combination of these two questions 
allows us to differentiate between parents who are 


A We cannot calculate a response rate because the 
roster information used to send surveys to school 
teachers/staff does not allow us to identify who 
is a pre-k teacher (they are self-identified on the 
surveys themselves). Therefore we cannot calculate 
a response rate. 


experiencing “OK” interactions with the director 
from those who are experiencing something harder 
to achieve— where the director seems to go above 
and beyond to connect with families. Construct 
validity can be confirmed by determining whether 
the item difficulties match the conceptual difficulty 
of those concepts (i.e., we confirm that a ques- 
tion that is empirically difficult also is one that is 
conceptually difficult to endorse). 

Internal validity can be confirmed by testing 
the unidimensionality— or that all questions 
are measuring a single construct— of the set 
of questions and by confirming the fit of the 
data to the model. The Rasch model calculates 
an expected response for each person to each 
item, and the degree to which people and 
items in the aggregate are acting in accordance 
with expectation produces measures of fit. Fit 
statistics included in the output help determine 
whether there are questions measuring a concept 
other than the one being assessed by the other 
questions in that measure (indicating that 
we should perhaps reject the presumption of 
unidimensionality). We may increase the internal 
validity of measures by removing questions not 
related to the concept being measured, or by 
adding questions that enhance the definition of 
the concept. We use the item fit statistics, along 
with correlations between questions, to verify 
that our measures only include questions that are 
measuring the degree to which people endorse a 
single, underlying concept. Analogously, a person 
with a poor fit statistic (“large misfit”) is likely 
someone who is responding in unexpected ways. 
We are likely to be skeptical of the measure from 
a person with a large misfit statistic; to adjust for 
this in analyses, we inflate the standard error of 
the person to reflect our uncertainty about that 
person’s measure score. 


B Office of Planning, Research & Evaluation, Adminis- 
tration for Children & Families, U.S. Department of 
Health & Human Services (2016). 

C Linacre (2015). 

D See http://consortium.uchicago.edu/sites/default/ 
files/uploads/survey/Overview of Anaiyses_0.pdf 
for an overview of the Rasch model and its benefits 
for survey development. 
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TABLE 1 

Rasch Reliabilities for the Five Essentials— Early Education Teacher Survey Pilot 


Measure 

Rasch Reliability 

Coefficient 

Number of Teachers 

Completing at Least 
One Question 

Within Measure* 3 

Essential: Effective Instructional Leaders 



Instructional Leadership 

0.85 

1,304 

Program Coherence 

0.82 

1,295 

Teacher Influence 

0.89 

1,298 

Teacher-Principal/Director Trust 

0.93 

1,324 

Essential: Collaborative Teachers 

Collective Responsibility 

0.92 

1,315 

Collective Use of Assessment Data 

0.86 

1,317 

Orientation to Innovation 

0.86 

1,504 

Quality Professional Development 

0.82 

1,294 

Reflective Dialogue 

0.87 

1,336 

School Commitment 

0.80 

1,319 

Socialization of New Teachers 

0.79 

1,334 

Teacher-Teacher Trust 

0.86 

1,322 

Teacher Collaboration 

0.87 

1,323 

Essential: Involved Families 

School/Center Welcoming to Families b 

0.74 

1,311 

Teacher Collaboration with Parents 

0.78 

1,319 

Teacher-Parent Trust 

0.89 

1,318 

Essential: Supportive Environment 

Attendance 

0.71 

1,291 

Child-Child Interactions 

0.80 

1,288 

Early Childhood Discipline 13 

0.58 

1,292 

Positive Learning Climate c 

0.78 

902 

Teacher Safety 

0.89 

1,320 

Essential: Ambitious Instruction 

Early Cognitive Development Instruction* 1 

0.87 

705 

Early Language and Literacy Instruction* 1 

0.87 

699 

Early Math Instruction* 1 

0.83 

723 

Early Social-Emotional Development Instruction* 1 

0.79 

744 

Preschool Pedagogical Practices 13 ’* 1 

0.75 

707 

Quality of Student Interactions' 3 

0.87 

901 


a Because each measure was only given to roughly half of the FACES sample (either 177 or 186 teachers; see the box titled 
Piloting the Surveys: Data Collection and Analyses'), the maximum number of possible respondents for each measure 
is either 1,330 or 1,339. The exception is Orientation to Innovation, which had questions on both versions for FACES 
administration; the maximum number of possible respondents for that measure is 1,516. 
b This measure has since been removed from the survey. 

c This measure was limited to a subset of teachers in CPS who identified themselves as teaching either a self-contained 
classroom ora subject-specific classroom. Only 64 percent of preschool teachers identified themselves as one of those, 
accounting for lower response numbers. 

d This measure was randomly assigned to teachers within CPS, accounting for lower response numbers. 
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2016 to see if their reliability improved with the 
revisions. Other characteristics of the measures 
also informed further refinement; for example, if 
the questions within a measure were not sensitive 
enough to differentiate teachers who strongly en- 
dorsed the concept, we created questions that were 
harder to endorse for testing in 2016. 

Results from the Five Essentials— Early Education 
parent survey are presented in Table 2. 51 As shown, 
the Rasch reliabilities for measures tested on our 
parent survey were more variable than those on 
the teacher survey. Recall that the parent survey 
was created anew while the teacher survey was 
largely adapted from existing surveys in use; thus 
it is not surprising that the measures were less 
reliable as they had less time in development. 


As with the teacher survey, we aim for measures 
to have reliabilities above 0.80, with acceptance 
of measures above 0.70. 

Because of the low reliabilities on four of the 
parent survey measures, we have made substan- 
tial adjustments to the survey and are testing 
those new revisions in our current validation 
study. For instance. Support for Kindergarten 
Transition was revised by keeping several of the 
pilot-tested questions, but developing several 
new questions that we believe will improve the 
reliability of the measure. Other measures, such 
as Parent Engagement and Quality of Engagement 
Opportunities, were re-conceptualized entirely. 
They originally had questions that focused on 
parent involvement in school or center events; 


TABLE 2 

Rasch Reliabilities for the Five Essentials— Early Education Parent Survey Pilot 


Measure 

Rasch Reliability 

Coefficient 

Number of Parents 

Completing at Least 

One Question 

Within Measure 

Essential: Effective Instructional Leaders 

Principal/Di rector- Pa rent Relations 

0.88 

245 

Essential: Involved Families 

Curriculum-Related Teacher Communication 

with Parents 

0.95 

229 

Family Engagement in Center/School 

0.68 

246 

General Teacher Communication with Parents 

0.91 

244 

Parent-Teacher Trust 3 

0.89 

124 

Quality of Engagement Opportunities 11 

0.61 

227 

Teacher Care and Responsiveness Towards Parents 

0.80 

245 

Essential: Supportive Environment 

Child-Child Interactions 

0.51 

249 

Support for Kindergarten Transition 15 

0.61 

175 

Program Orientation Towards Early Education 

0.84 

246 


a Two versions of this measure were tested, and the reliability is only shown for one version. This accounts for the lower 
response numbers. 

b This measure depended on a preceding question. Support for Kindergarten Transition was only asked of parents who 
reported that their child was transitioning into kindergarten the following year; Quality of Engagement Opportunities 
was only asked of parents who reported Engagement in the Center/School (preceding measure). 


51 Table 2 lists the measures under the essential we the relational structure of measures to essentials as 

hypothesize that they will load onto. We will be testing part of our 2016 validation study. 
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TABLE 3 

Revisions to Five Essentials— Early Education Parent Survey for 2016 Testing 


Piloted Measures Kept for 2016 Testing 

New Measures Developed for 2016 Testing 

Principal/Director-Parent Relationships 

• Program Orientation towards Early Education 

• Support for Kindergarten Transition 

- Teacher Communication with Parents 

• Including Parents as Partners 

• Parent Influence on the Program 

• Social-Capital Building of Parents 

• Staff Care of Parent as Person 

• Teachers' Interactions with Children 


however, in making revisions, we heeded the advice 
of experts who encouraged us to ask questions more 
aligned with what how the held conceptualizes key 
aspects of family engagement. This led us to shift 
the focus toward whether parents felt they had 
influence on the program, whether they felt they 
were partners in educating their child, and whether 
there were supports provided to build their own 
social capital. Researchers dedicated to under- 
standing family engagement practices have found 
that these types of connections between families 
and programs are most impactful on both family 
and child outcomes. 52 There were other revisions 
made as well, including taking questions in some 
measures and developing them out into multiple 
new measures. For instance, although Teacher Care 
and Responsiveness had high reliability, there were 
other indications that it was capturing more than 
one construct. Therefore, some of those items were 
incorporated into the newly developed Staff Care 
of Parent as Person and Social- Capital Building of 
Parents. With the substantial revisions conducted 
since the pilot study. Table 3 outlines which mea- 
sures were retained on the parent survey for a new 
round of testing in 2016, as well as a listing of new 
measures we created after the completion of the 
pilot study. 


A Validation Study is 
Currently Underway 

With revised and new measures created, UChicago 
Consortium and the Ounce are currently test- 
ing external validity of the surveys in a valida- 
tion study. The box titled Definitions of Five 
Essentials— Early Education Measures being 
Tested in 2016 on p.19 gives an overview of the 
current measures we are testing and a descrip- 
tion of the concepts they intend to capture. Over 
the course of the 2015-16 school/program year, 
the research team collected data to test whether 
program-level variations in responses to the Five 
Essentials— Early Education surveys are related 
to pre-k program outcomes we believe matter, 
specifically teacher-child interactions within 
the classroom and child development outcomes. 
Analyses on this round of data collection will also 
help us further refine the surveys. For example, we 
plan to decrease the overall number of questions 
and measures to shorten the survey, thus reduc- 
ing burden on teachers and parents. We also plan 
to use newly collected survey data to empirically 
identify which measures from each of the surveys 
cluster together under each of the five essential 
supports. 53 


52 e.g., Halgunseth (2009); Mapp & Kuttner (2013). 

53 We also will be testing the assumption that there are 
five essential supports in early education. It is theoreti- 
cally possible that a subset of the five essential sup- 
ports identify early education programs that display 
positive classroom practices and child outcomes. 
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Definitions of Five Essentials— Early Education Measures 
being Tested in 2016 


Our 2016 data collection efforts focus on testing survey measures that capture teachers' and parents’ ex- 
periences and beliefs about their (or their child’s) preschool center or school. This box provides definitions 
of each construct being tested on the 2016 validation study versions of the surveys. Those marked with a 
(T) represent measures on the teacher survey and those marked with a (P) are on the parent survey. The 
survey measures are listed under the five essential supports, based on how we hypothesize each measure 
will load onto each construct. We will test the relational structure of measures to essential supports as 
part of our 2016 study. 

Effective Instructional Leaders 

Instructional Leadership* (T): The principal/director is an active and skilled instructional leader 
who sets high standards for teaching and children's learning. 

• Principal/Director-Parent Relationships (P): School/Center leaders actively reach out to and are 
responsive to parents. 

• Program Coherence* (T): School/center programs are coordinated and consistent with its goals 
for children’s learning. 

• Program Orientation towards Early Education (P): The school/center approaches their work 
with children as early educators responsible for preparing children for kindergarten. 

• Teacher Influence* (T): Teachers have influence on a broad range of decisions regarding school/ 
center policies and practices. 

• Teacher-Principal/Director Trust* (T): Teachers and school/center leaders share a high level 
of mutual trust and respect. 

Collaborative Teachers 

• Collective Responsibility* (T): Teachers share a strong sense of responsibility for children’s 
development and learning, school improvement, and professional growth. 

• Collective Use of Assessment Data* (T): Teachers review child assessment data with others in 
the school/center. 

• Orientation to Innovation* (T): Teachers are continually learning and seeking new ideas, have a 
“can do” attitude, and are encouraged to try new ideas in their teaching. 

Quality Professional Development* (T): Professional development is rigorous, sustained, and 
focused on children’s learning. 

• Reflective Dialogue* (T): Teachers talk with and learn from one another about instruction and 
children’s learning. 

• School Commitment* (T): Teachers are deeply committed to the school/center. 

• Socialization of New Teachers* (T): New teachers are included in the professional community 
and are given helpful feedback on their instructional practices. 

• Teacher-Teacher Trust* (T): Teachers are supportive and respectful of one another, personally and 
professionally. 

• Teacher Collaboration* (T): Teachers observe each other’s practice and work together to review 
assessment data and develop instructional strategies. 
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DEFINITIONS OF FIVE ESSENTIALS — EARLY EDUCATION MEASURES . ..CONTINUED 


Involved Families 

• Including Parents as Partners (P): Staff respect parents’ inputs and concerns as knowledgeable 
partners in supporting their child’s learning and development. 

Parent Influence* (T): Teacher perceptions of whether the school/center involves parents of various 
backgrounds in making programming decisions. 

• Parent Influence on the Program (P): Parent perceptions of whether the school/center actively 
includes parents of various backgrounds in program improvement efforts. 

• Parent Involvement (T): Teacher perceptions of parents as active participants in their child’s 
experience in the school/center. 

Teacher Communication with Parents (P): Teachers provide specific feedback on children’s learning 
and development to parents. 

Teacher Collaboration with Parents (T): Teachers shape their teaching practices based on input from 
parents and provide parents with information on how to support their child’s learning at home. 

• Teacher-Parent Trust* (T): Teachers feel supported and trusted by parents. 

Supportive Environment 

• Attendance (T): Teachers review attendance data and use it to reach out and provide supports to families. 

• Child-Child Interactions (T): Children interact with each other in positive ways. 

• Positive Learning Climate (T): Children act within the classroom with comfort and enthusiasm to learn. 

• Social-Capital Building of Parents (P): Staff help families develop social capital through connections 
with other adults and resources. 

• Staff Care of Parent as Person (P): School/center staff extend expressions of care for parents as individuals, 
beyond the interest in their child. 

Support for Kindergarten Transition (P): Staff support parents in planning for the transition to kindergarten. 

• Teachers’ Interactions with Children (P): Teachers express care and respect for children in their classroom. 

• Teacher Safety* (T): Teachers report little or no disorder in the hallways, physical conflict among 
students/children, vandalism, robbery or theft, and threats of violence against teachers. 

Ambitious Instruction 

• Early Cognitive Development Instruction (T): Teachers provide opportunities for children to learn, 
practice, and apply early cognitive development skills, such as formulating predictions, comparisons, 
and explanations for how things work. 

Early Literacy and Language Instruction (T): Teachers provide opportunities for children to learn, 
practice, and apply critical early literacy and language skills. 

• Early Math Instruction (T): Teachers provide opportunities for children to learn, practice, and apply 
critical early mathematics concepts and skills. 

Early Social-Emotional Instruction (T): Teachers provide opportunities and for children to learn, 
practice, and apply early social-emotional skills. 

Quality of Student Interactions (T): Children interact in ways that specifically support each other’s 
academic and social-emotional learning and development. 

• These measures come from the existing K-12 5Essentials survey, either exactly or slightly modified for appropriate use in early 
education settings. 
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Next Steps in Developing, 

Testing, and Implementing 
the Five Essentials— Early 
Education Surveys 

As we continue our work, we will investigate both 
the teacher and parent surveys to understand 
whether they should be used across different set- 
tings and with different populations of survey-tak- 
ers. This includes examining possible differences 
in the (1) non-response rates to questions and mea- 
sures (missingness); (2) rates of misfitting people 
(people who did not respond to the set of questions 
within a measure in an expected fashion); and 
(3) Differential Item Functioning (DIF) between 
groups. DIF analysis allows us to look at respon- 
dents in two groups and ask: Are there particular 
survey items (questions) that are easier or harder 
to endorse by teachers/parents in one group than 
another group, given similar overall beliefs or ex- 
periences with the construct being asked about? 54 
For example, the Five Essentials— Early Education 
surveys were developed to be used in both school- 
based and center-based settings. However, it is pos- 
sible that the experiences in those two settings are 
different enough that some survey questions might 
work well with teachers or parents in one setting 
but not in the other. Our analyses can test for these 
differences, helping us determine whether the same 
survey can be used across both settings or if there 
is a need to create two versions. We similarly will 
examine whether the surveys function comparably 
among English- and Spanish-speaking parents. 

During the 2017-18 school/program year, once 
the current validation study is complete, we will 
work closely with a small set of schools and center- 


based programs to help us prepare for broader 
implementation and use of the surveys in the field. 
We have two goals for this implementation pilot. 
The first is to test and finalize methods of survey 
administration, scoring, and reporting. The sec- 
ond is to design and refine professional resources 
and tools that enable leaders, teachers, and other 
staff to use the survey data effectively in their set- 
ting. Our aim is to have the Five Essentials— Early 
Education surveys align with the K-12 5Essentials 
surveys, while simultaneously attending to the 
nuances of the early education field. Having a set 
of surveys that are conceptually related to each 
other can support current alignment efforts across 
pre-k and elementary grades. 55 However, we will 
consider various options for scoring the early edu- 
cation version, which may or may not be different 
from the K-12 version. Currently, schools taking 
the K-12 5Essentials surveys receive reports com- 
paring their results to those of other schools. But 
for the Five Essentials— Early Education surveys, 
we have two settings— schools and community- 
based centers— and need to consider the best way 
of providing comparison information. Should 
programs receive comparison information only 
for early education programs in the same setting? 
Or, should programs receive comparison informa- 
tion for all early education programs regardless 
of the setting? These are examples of questions 
we will grapple with as we transition from the 
development phase of survey creation to the im- 
plementation phase. By fall 2018, we plan to have 
the Five Essentials— Early Education surveys and 
corresponding professional resources and tools 
ready for implementation and use in the field. 


54 There can still be true differences in the levels of endorse- 55 e.g., Kauerz & Thorman (2011); Nyhan (2015, June 24); 
ment between groups without seeing significant and Sadowski (2006, October), 

large DIFs; rather, this tests for individual item function- 
ing within the measure across groups. 
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Conclusion 

The early education field has a long tradition of 
rigorous assessment of classroom-level structures 
and instructional quality and their relationships 
to children’s outcomes. The goal for the Five 
Essentials— Early Education surveys is to comple- 
ment these efforts and provide a critical missing 
piece— reliable and valid survey data on the organi- 
zational conditions of early education programs. 

Within schools and centers, our hope is that 
data from the Five Essentials— Early Education sur- 
veys will provide leaders and staff with actionable 
information to focus their attention on strengthen- 
ing the organizational supports that enhance and 
improve teachers’ complex work with children and 
families. For policymakers and program leaders, we 
envision that the Five Essentials— Early Education 
surveys can broaden definitions of quality and 
transform thinking about how instructional im- 
provement is generated. Specifically, we hope that 
program staff and leaders can use these surveys to 
identify organizational strengths and weaknesses 
and engage in collaborative conversations about 
implementation of improvement efforts. 


Practitioners and leaders across the nation 
have cautioned carefully— and we agree— that in 
order for these surveys to achieve their intended 
goals, it is imperative that they not become an- 
other measure used in accountability systems; 
teachers and parents need to know that they can 
express their voices without the threat of regula- 
tion, or worse, loss of funding for their programs. 
Without becoming an accountability metric, these 
surveys can be used by practitioners and system 
leaders to improve organizational-level processes 
that support those high-quality practices assessed 
by existing early childhood quality rating and 
improvement systems (QRIS). 

Lastly, our hope is that the Five Essentials— 
Early Education surveys will support alignment 
efforts between early and elementary educators. 
When used together, the Five Essentials— Early 
Education surveys and the K-12 5Essentials 
surveys have the potential to provide a common 
lens, language, and metric for understanding 
key organizational conditions for instructional 
improvement across the educational continuum. 
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